Testing Environment

We conducted tests using the imagenet_train dataset, which consists of 1,281,167 images. These images were embedded using a 2048-dimension doubao-embedding model, resulting in a dataset size of approximately 140GB after converting to lance.

• Single-machine index building

  • Machine: 4 cores, 16GB RAM
  • Storage: object-store for original dataset

• Distributed setup

  • Topology: 1 head node + 4 worker nodes
  • Each node: 4 cores, 16GB RAM
  • Purpose: use very cheap machines to extend overall runtime and evaluate performance improvements of distributed index building on low-config machines

• Index building parameters

  • num_partitions = 1000
  • Data split into 253 fragments
  • Distributed processing: 4 workers in parallel

Testing Results

• Unit: m = minute

IVF_SQ Group (Tested on S3)

IVF_FLAT Group

IVF_PQ Group

Overall Observation

Since it works on S3, network speed and S3 I/O may vary during the process. Sometimes it can take hours, but mainly due to the reasons below.

The distributed index was built using 4-core, 16GB machines (4 workers in total). We also checked if performance could scale linearly. The main reasons for the performance acceleration are:

1. Avoiding backpressure throttle exceeded in IO on a single machine.
   We observed that when building an index on a low-performance machine, the lance process kept reporting:

   "backpressure throttle exceeded in IO"

2. Distributed assignment allows using CPU resources from worker nodes to parallelize the process.

Conclusion

The test was carried out with a 4-core, 16GB machine and an S3 object-store. The results clearly show that the distributed setup significantly outperforms the single-machine setup in terms of index building time, especially for the Distribute Builder Time metric, for scaling linearly or better.

We can estimate speedup as:
For example, IVF_SQ total time​≈4.03×
Nearly scaling linearly

Limitation

Since PQ (Product Quantization) depends on:

• global training of the Inverted File (IVF), and
• PQ codebook training,

both of which are carried out on a single machine, if the training of the PQ codebook takes a long time, the acceleration effect of IVF-PQ may not be satisfactory. However, it is more suitable for a larger number of tables.

The most well-balanced and superior method appears to be IVF_SQ (Inverted File with Scalar Quantization), as this method can generally guarantee good recall and support distributed parallelism.

@chenghao-guo can you make sure we add this to documentation? Another thing missing is we probably should propagate GPU configs like accelerator="cuda", which would be a key to take advantage of Ray's GPU native support.

@chenghao-guo can you make sure we add this to documentation? Another thing missing is we probably should propagate GPU configs like accelerator="cuda", which would be a key to take advantage of Ray's GPU native support.

Hi Jack, thanks a lot for the review. I’ll update the documentation accordingly in the md file.
For accelerator="cuda", I’ll also verify whether it works as expected before proceeding further.